URSSI Analysis

Completion Rate

## There are 735 Qualtrics surveys marked finished.

## There are 459 Qualtrics surveys marked not finished.

## There were a total of 1194 participants. 
##  459 participants, 38.44 % of participants, did not finish the survey.

Which profile matches the work you perform?

Demographics

What type of organization do you work for?

US States

List of Countries

Countries Breakdown

##              
##               Combination Developers Researchers
##   Australia             1          0           0
##   Canada                5          1           2
##   China                 1          0           0
##   France                3          1           1
##   Germany               9          1           1
##   India                 0          1           1
##   Iran                  1          0           0
##   Japan                 3          0           0
##   Netherlands           1          0           1
##   Norway                1          0           0
##   Romania               1          0           0
##   South Korea           1          0           0
##   Spain                 1          0           0
##   Switzerland           1          2           0
##   Turkey                0          0           1
##   UK                    3          1           2
##   USA                 486         73         431

Job Titles

Other Job Titles

In which discipline(s) do you perform research?

Age

How many years have you worked in research?

Gender

How often do you work on projects that have the following team size?

Software Development Practices

To what extent was the software targeted at each of these groups of users?

What percent of your time did you actually spend in the following activities (total should be 100%)?

What percent of time would you have liked to have spent in each activity (total should be 100%)?

Comparative

Due to the large difference in group sizes (Combination having n = 544 and Developers having n = 89), ANCOVA with type 3 sums of squares is run for checking differences between groups. When grouping is ignored, a paired t-test is used instead.

T-tests show that the Current(Q3.3) and Ideal(Q3.4) questions all have significantly different means when grouping is ignored.

ANCOVA shows that there are 2 activities Developers and Combination researchers differ significantly on.

Developers are more likely to express wanting to spend more time on maintenance than those in the Combination grouping. (Q3.3_6 and Q3.4_6)

On the other hand, those in the combination group are more likely to want to allocate more time to meetings in their ideal scheduling than Developers. (Q3.3_8 and Q3.4_8)

Statistical Tests

Requirements

## Anova Table (Type III tests)
## 
## Response: Ideal
##                Sum Sq  Df  F value  Pr(>F)    
## (Intercept)     332.1   1   6.5562 0.01076 *  
## Current        7195.0   1 142.0267 < 2e-16 ***
## Group            13.0   1   0.2562 0.61298    
## Current:Group   130.7   1   2.5791 0.10896    
## Residuals     23860.7 471                     
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = -1.1336, df = 474, p-value = 0.2575
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -1.0588084  0.2840716
## sample estimates:
## mean of the differences 
##              -0.3873684

Software

## Anova Table (Type III tests)
## 
## Response: Ideal
##               Sum Sq  Df  F value    Pr(>F)    
## (Intercept)     3012   1  25.4705 6.428e-07 ***
## Current        11943   1 100.9871 < 2.2e-16 ***
## Group              0   1   0.0006    0.9807    
## Current:Group      7   1   0.0582    0.8094    
## Residuals      55700 471                       
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = -10.088, df = 474, p-value < 2.2e-16
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -6.009098 -4.049849
## sample estimates:
## mean of the differences 
##               -5.029474

Coding

## Anova Table (Type III tests)
## 
## Response: Ideal
##               Sum Sq  Df  F value    Pr(>F)    
## (Intercept)     7640   1  25.9775 5.011e-07 ***
## Current        32202   1 109.4863 < 2.2e-16 ***
## Group              1   1   0.0037    0.9516    
## Current:Group    159   1   0.5404    0.4626    
## Residuals     138528 471                       
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = -4.7079, df = 474, p-value = 3.292e-06
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -5.466622 -2.247062
## sample estimates:
## mean of the differences 
##               -3.856842

Testing

## Anova Table (Type III tests)
## 
## Response: Ideal
##               Sum Sq  Df F value    Pr(>F)    
## (Intercept)      775   1  9.4979  0.002178 ** 
## Current         4345   1 53.2314 1.267e-12 ***
## Group             29   1  0.3565  0.550762    
## Current:Group      4   1  0.0442  0.833504    
## Residuals      38448 471                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = 3.2628, df = 474, p-value = 0.001183
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.5945622 2.3949114
## sample estimates:
## mean of the differences 
##                1.494737

Debugging

## Anova Table (Type III tests)
## 
## Response: Ideal
##                Sum Sq  Df F value    Pr(>F)    
## (Intercept)     382.4   1 10.9410  0.001013 ** 
## Current        1155.2   1 33.0541 1.613e-08 ***
## Group             4.0   1  0.1151  0.734613    
## Current:Group     4.1   1  0.1169  0.732563    
## Residuals     16460.6 471                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = 16.386, df = 474, p-value < 2.2e-16
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  6.090185 7.749815
## sample estimates:
## mean of the differences 
##                    6.92

Maintenance

## Anova Table (Type III tests)
## 
## Response: Ideal
##                Sum Sq  Df F value    Pr(>F)    
## (Intercept)     397.1   1 14.2125 0.0001840 ***
## Current        2597.9   1 92.9719 < 2.2e-16 ***
## Group            95.8   1  3.4295 0.0646685 .  
## Current:Group   414.6   1 14.8392 0.0001333 ***
## Residuals     13160.9 471                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

##  Group       Current.trend     SE  df lower.CL upper.CL
##  Combination         0.228 0.0361 471    0.157    0.299
##  Developers          0.531 0.0699 471    0.394    0.668
## 
## Confidence level used: 0.95

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = 5.41, df = 474, p-value = 1.002e-07
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  1.199845 2.568576
## sample estimates:
## mean of the differences 
##                1.884211

Documentation

## Anova Table (Type III tests)
## 
## Response: Ideal
##                Sum Sq  Df F value    Pr(>F)    
## (Intercept)    1701.4   1 34.0199 1.015e-08 ***
## Current        3011.5   1 60.2172 5.320e-14 ***
## Group            72.2   1  1.4430    0.2303    
## Current:Group   100.8   1  2.0147    0.1564    
## Residuals     23555.1 471                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = -5.6254, df = 474, p-value = 3.168e-08
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -2.619071 -1.263035
## sample estimates:
## mean of the differences 
##               -1.941053

Meetings

## Anova Table (Type III tests)
## 
## Response: Ideal
##                Sum Sq  Df F value    Pr(>F)    
## (Intercept)     238.5   1  3.5564  0.059930 .  
## Current        4864.6   1 72.5440 < 2.2e-16 ***
## Group           199.3   1  2.9719  0.085376 .  
## Current:Group   709.5   1 10.5806  0.001225 ** 
## Residuals     31584.1 471                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

##  Group       Current.trend     SE  df lower.CL upper.CL
##  Combination         0.576 0.0307 471    0.516    0.636
##  Developers          0.258 0.0929 471    0.075    0.440
## 
## Confidence level used: 0.95

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = 7.4989, df = 474, p-value = 3.195e-13
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  2.572772 4.399859
## sample estimates:
## mean of the differences 
##                3.486316

Training

## Anova Table (Type III tests)
## 
## Response: Ideal
##                Sum Sq  Df F value    Pr(>F)    
## (Intercept)     924.4   1 14.2850 0.0001773 ***
## Current        1301.3   1 20.1089   9.2e-06 ***
## Group           184.3   1  2.8479 0.0921556 .  
## Current:Group   167.3   1  2.5857 0.1085062    
## Residuals     30479.1 471                      
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = -3.2642, df = 474, p-value = 0.001177
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -2.1348580 -0.5304052
## sample estimates:
## mean of the differences 
##               -1.332632

Other

## Anova Table (Type III tests)
## 
## Response: Ideal
##               Sum Sq  Df  F value Pr(>F)    
## (Intercept)      261   1   1.5343 0.2161    
## Current        33261   1 195.2458 <2e-16 ***
## Group             91   1   0.5328 0.4658    
## Current:Group     11   1   0.0617 0.8039    
## Residuals      80236 471                    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

## 
##  Paired t-test
## 
## data:  Pair(Current, Ideal)
## t = -2.06, df = 474, p-value = 0.03995
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -2.41870123 -0.05708825
## sample estimates:
## mean of the differences 
##               -1.237895

Which of the following types of testing did you employ?

Did you release the software under an open-source license?

Development Infrastructure and Tools

In the last 5 years, how much experience do you have with each of the following types of projects

How well do current tools support each of the following activities?

Training

Have you received training for software development?

Where?

Do you believe there are sufficient opportunities available for exploring or obtaining new software skills?

Do you have sufficient time to take the types of training you need to be successful?

For each of the following types of training (languages, development techniques, and project management), which mode of delivery do you prefer? Please select your top two modes of delivery preferences for each of the training types.

Funding/Institutional Support

Have you ever included costs for software development in a funding proposal?

Select all costs for software development in your funding proposals?

What percent of the funding for your software work comes from each of the following?

Does your institution provide support for your research software development activities in the following ways

Does your institution provide / support the research software development you need in the following ways

What are the most significant gaps in your current institutional / organizational support of software related research?

TODO

Career Paths

In your institution, which of the following positions are available for people whose primary role is to develop research software?

Is there opportunity for career advancement (e.g., tenure, more senior positions) for people whose primary role is to develop research software at your institution?

With regard to hiring and maintaining a highly qualified software development staff, how important are the following concerns?

In your opinion: How important were the following concerns when you were hired into your current position?

Credit for Software Contributions

When you write a paper and the work being described uses software, how often do you use each of the following approaches to mention the software? - Cite paper about the software

How often do you currently receive the following types of credit for your software contributions?

For your job role, does your institution allow software contributions to be considered in performance reviews or promotion cases?

How important do you believe software contributions are for your own performance review or promotion case?

Diversity/Inclusion

How well does your project do the following

What are the challenges you face in creating / maintaining a culture of inclusion?

Does your project do any measurement, tracking, or analysis of developer or user communities - with regard to diversity and/or inclusion?

Follow-up

Are you willing to answer a few more questions about any of the following areas?

Software Development Practices Follow Up

Maintenance is important for the long-term sustainability of a software project. For the following activities that are considered part of software maintenance, indicate what type of support you need for each activity

Overall, regardless of where you spend the most time, which of the following aspects of software development are more difficult than you think they should be?

How often do you employ the following practices when developing software?

How often do you perform each of the following practices when developing software?

How often do you document the following information when developing software?

Development Infrastructure and Tools Follow Up

Do you use use a version control system?

How often do you use each of the following for version control systems.

On average, how frequently do you check-in or commit code to a version control system?

Do you use continuous integration?

Training Follow Up

Please list any specific software-related training activities that you have found helpful (either the name, a link, or the topic).

What software-related topics would you or your team like to receive training about?

TODO

Funding Institutional Support Follow Up

In general, do you think you have sufficient funding to support software development activities in your research?

In particular, how well does your current funding support the following activities?

If you develop new software, does your current funding support the following activities

The biggest financial burdens I face in using or developing software for my research are:

If you were given a budget of $100 million to support research software development and maintenance in the USA how would you choose to spend it (i.e. to which types of activities would you allocate money and how much)? Please allocate your budget in millions of dollars. Responses need to total to $100.

Career Path Follow Up

Do other members of your organization or institution use your software?

Have other members of your organization contacted you about developing software for them?

Rate the importance of each of the following factors when deciding upon accepting a new job position or promotion.

Credit for Software Follow Up

If you use open source software not developed by you, how important is the number of downloads for your decision to use it?

If you use open source software not developed by you, how important is the number of contributors for your decision to use it?

If you use open source software not developed by you, how important is the presence of an academic publication about the software for your decision to use it?

Diversity and Inclusion Follow Up

Considering the software project that you spend the most time working on, does this project have a diversity / inclusion statement or policy?

You responded that the software project that you spend the most time working on has a diversity / inclusion statement or policy. In the box below, please provide a link to that statement or policy.

Considering the software project that you spend the most time working on or with, does this project have a code of conduct?

You stated that the software project that you spend the most time working on or with has a code of conduct. In the box below, please provide a link to that code of conduct.

Which of the following inclusive practices and policies do you need support in creating for projects that focus on or include software development? Please mark all that apply.